Cap max RetryableAction wait time/timeout.#74940
Merged
henningandersen merged 4 commits intoelastic:masterfrom Aug 5, 2021
Merged
Cap max RetryableAction wait time/timeout.#74940henningandersen merged 4 commits intoelastic:masterfrom
henningandersen merged 4 commits intoelastic:masterfrom
Conversation
RetryableAction uses randomized and exponential back off. If unlucky, the randomization would cause a series of very short waits, which would double the bound every time, risking a subsequent very long wait. Now randomize between [bound/2, bound[. Closes elastic#70996
henningandersen
added
>bug
:Distributed/CRUD
elasticmachine
added
Team:Distributed
Collaborator
|
Pinging @elastic/ml-core (Team:ML) |
Collaborator
|
Pinging @elastic/es-distributed (Team:Distributed) |
davidkyle
approved these changes
Jul 6, 2021
Member
davidkyle
left a comment
davidkyle
left a comment
There was a problem hiding this comment.
ML changes LGTM
Consider renaming the calculateDelay(long) function to calculateMaxDelay(long) or calculateMaxDelayBound(long) as the logic to determine the actual delay passed to threadPool.schedule is in the onFailure method.
...n/ml/src/main/java/org/elasticsearch/xpack/ml/utils/persistence/ResultsPersisterService.java
Outdated
Show resolved
Hide resolved
benwtrent
approved these changes
Jul 12, 2021
Contributor
|
Will this PR fix timeouts like this one? https://gradle-enterprise.elastic.co/s/qvkqnuh5ocffw |
Contributor
Author
I am afraid not, the symptoms of the failing |
…s/persistence/ResultsPersisterService.java Co-authored-by: David Kyle <david.kyle@elastic.co>
henningandersen
added
the
auto-backport
Collaborator
💔 Backport failed
To backport manually run: |
henningandersen
removed
the
auto-backport
henningandersen
added a commit
to henningandersen/elasticsearch
that referenced
this pull request
Aug 5, 2021
RetryableAction uses randomized and exponential back off. If unlucky, the randomization would cause a series of very short waits, which would double the bound every time, risking a subsequent very long wait. Now randomize between [bound/2, bound[. Closes elastic#70996
henningandersen
added a commit
that referenced
this pull request
Aug 5, 2021
RetryableAction uses randomized and exponential back off. If unlucky, the randomization would cause a series of very short waits, which would double the bound every time, risking a subsequent very long wait. Now randomize between [bound/2, bound[. Closes #70996
henningandersen
added a commit
that referenced
this pull request
Aug 5, 2021
RetryableAction uses randomized and exponential back off. If unlucky, the randomization would cause a series of very short waits, which would double the bound every time, risking a subsequent very long wait. Now randomize between [bound/2, bound[. Closes #70996
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RetryableAction uses randomized and exponential back off. If unlucky,
the randomization would cause a series of very short waits, which would
double the bound every time, risking a subsequent very long wait. Now
randomize between [bound/2, bound[.
Closes #70996
This fixes
testAckedIndexingbecause it ran into a final very long wait time for one of the retries.I am not necessarily fixed on this solution, though it seems like a step in the right direction. An
alternative could be to cap the final wait time to meet the timeout, but it does add more deterministic/less random
behavior and the solution here should be good enough.